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ABSTRACT 

A study tested an indirect assessment instrument in 
examining the cognitive component of communication competence. The 
instrument was administered to 36 seniors graduating in speech 
communication and mass communication at the University of 
Missouri-Kansas City. Findings revealed that test performance was 
mediated by overall grade point average, 'non-traditional methods of 
instruction (i.e., internship), and student enrollment patterns 
(i.e., number of >ears needed to complete the degree program and 
full-time versus part-time employment) . Should additional testing 
confirm the findings of the study, indirect tests may be administered 
with reasonable confidence as diagnostic measures for placing 
students in the proper courses. Several lines of research may aid in 
more effectively administering indirect tests at the departmental 
level. (SR) 
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The study tested an indirect assessment instrument in examining the 
cognitive component of communication competence. Findings 
revealed that test performance was mediated by overall grade point 
average, non-traditional methods of instruction (i.e., internship), and 
student enrollment patterns (i.e., number of years needed to 
complete the degree program and full-time vs. part-time 
employment) The study concludes by discussing the role of indirect 
tests in relation to direct-based behavioral assessments of 
communication competence. Several lines of research are suggested 
so that indirect tests may be more effectively administered at the 
departmental level. 

♦Michael R. Neer is an Associate Professor of Communication Studies 
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INTRODUCTION 

The University of Missouri syctem, in October 1987, required that all 
four of its campuses begin conducting annual assessments of all seniors. 
The assessment was mandated by the Governor as a means of determiniing how 
effectively the university was fulfilling its mission of providing students 
with a quality education. The assessment was therefore designed to measure 
whether students have received sufficient instruction in areas defined as 
essential to a well-rounded education by each academic unit of the university. 
The purpose of this study is to report findings for the assessment test de- 
veloped in communication at the Kansas City campus of the University of 
Missouri system. 

At the Kansas City campus (UMKC), assessment of basic English proficien- 
cy has been in operation for several years. Students who fail to achieve at 
least a grade of "B" in the two' university-required composition courses are 
then required to take the university's english proficiency test administered 
by the English department. However, with the state-mandated assessment, uni- 
versity-wide testing was required on two levels: (1) individualized assess- 
•ments developed by each academic unit within the university, and (2) 
standardized testing in general education through objective testing of all 
graduating seniors' abilities in problem-solving, clarifying values, func- 
tioning within social institutions, using science and technology, and using 
the arts. The standardized test used to assess these areas was the College 
Outcome Measures Project (COMP) developed by the American College Testing 
program. 

Assessment is by no means a novel idea or area of research within the 

3 



Assessment 
3 

conimunication discipline. As early as 1978, an issue of Cocmunication Edu- 
cation was devoted to communication literacy (see volume 31, 1978), and the 
first systematic review and evaluation of current assessment tests was re- 
ported in the text Assessing Functional Communication (Larson, Backlund, 
Redmond, & Barbour, 1978). The most ambitious asssessment test of communi- 
cation competence developed to date is perhaps the CCAI (Communication Com- 
petency Assessment Instrument) (Rubin, 1980, 1981, 1982, & 1985). The CCAI 
has demonstrated reliability and validity as an assessment 

of behavioral competence in communication. The instrument assesses communi- 
cation competence along four dimensions of funcnional competence:, communi- 
cation codes, oral message evaluation, basic speech communication skills, 
and human relations. 

The assessment instrument developed for this present study is an in- 
direct-based evaluation similar to that developed by Scafe and Siler (1979). 
Thus, the assessment test was a written objective test and not a direct test 
involving instructor-based observation of communication performance. Al- 
though indirect tests do not provide a means of identifying behavioral de- 
ficiencies in communication, they may he used to measure cognitive communi- 
cation ability as an underlying component of communication competence. For 
instance, a recent study (Rubin, 1985) reported that scores on a self-report 
scale of communication knowledge correlated at .52 with the behavior-based 
CCAI. These findings suggest that knowledge of how to communicate, in part, 
' predicts how one actually communicates. Vhile cognitive ability may not be 
used as a singular predictor of communication competence, indirect tests 
may be appropriate for diagnosing cognitive deficiencies contributing to 
behavioral incompetence. 
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The communication studies department at the Kansas City campus develop- 
ed its assessment test for diagnostic purposes. That is, we were concerned 
with developing a placement test for incoming transfer students who may have 
completed the required corts at other four-year institutions and two-year 
community colleges. The test also was intended for eventual use in diag- 
nosing students' cognitive competence relevant to advanced couraework in 
public speaking, speech and mass communication. Although enrollment would 
not be d.jnied on the basis of assessment scores, the assessment would be 
used to determine if basic theoretical principles learned in the required 
core courses need to be reintroduced before proficiency may be attained in 
the advanced courses. 
METHOD 

The department of Communication Studies consists of two areas of em- 
phasis: speech communication and mass communication. Although each empha- 
sis is intended to fulfill uniquely different educational objectives, all 
graduating seniors are expected to demonstrate proficiency in basic com- 
munication competencies. These include two required core courses in oral 
competence (public speaking) and basic communication theory. These two 
courses along with introduction to mass communication and introduction 
to media writing represent two additional competencies essential for media 
professionals. Collectively, all four courses comprised the assessment 
test evaluated in this report. ^ 
Test Constructio n 

The four-part test consisted of -multiple choice items (85 percent of 
the test) and one short essay question on principles of effective media 
writing. All four parts were equally weighed at 30 points or a 120-point 
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total. The test was constructed by a committee of faculty who regularly 
instruct the four courses. A pool of over 200 items was initially review- 
ed and eventually narrowed to 120 by committee consensus. Items were se- 
lected on the basis of key elements of instruction, established course ob- 
jectives, and on the basis of recurrent themes discussed in lecture and in 
the texts, ^ In constructing the test, syllabi were reviewed so that ques- 
tions would match the major learning objectives set in each course. The 
review of textbooks followed a similar procedure in question selection. 
That is, only those chapters or sections of a text emphasized in class lec- 
ture and course examinations were included in the assessment test. All 
texts in these courses have been regularly used over, a four-year period 
thus ensuring a consistent testing focus. Consistency was further en- 
sured since each course has been regularly taught by no more than two in- 
structors, each of whom have developed a common syllabus for the course. 
Test Design 

The two required courses focus on basic principles of public speaking 
(e.g., message preparation, audience analysis, speaker credibility, thesis 
development, and speech organization) and communication theory (e.g., per- 
ception, language processes, nonverbal communication, relational develop- 
ment, and conflict). The mass communication core courses focus on basic 
mass communication theory (e.g., media history, media impact, first amend- 
ment issues, media structure, and media news and advertising processes) 
and principles of media writing (e.g., media writing style relating to news, 
advertising, press releases, and other forms of writing for the media). 

Approximately one-third of each test component included items that re- 
quired synthesis and application of course content as opposed to simple 
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recall and recognition. The oral competence component required students 
to analyze speaking situations and select the most effective means of de- 
signicg an orally-communicated messages. For instance, students were pro- 
vided sample speaking situations and required to select the most appro- 
priate forms of support (i.e., testimonial, statistical, anecdotal); or 
organizational format most appropriate for structuring message content. 
Other items required correct identification of organizational formats and 
supporting materials. Thus, the public speaking component targeted several 
criteria listed as essential in assessment testing, including identifying 
main points, expressing and defending evidence, and organizing clear mess- 
ages (see for example: Bassett, Whittington, & Staton-Spicef , 1978). 

Several of the basic communication theory items also required students 
to analyze underlying causes of communication conflict in case study situ- 
ations and select possible remedies. Half of the communication- theory items 
focused on several components of communication competence including empathy, 
behavioral flexibility, and interaction management (see for example: Lar- 
son, 1978; Weimann, 1978). Remaining items covered material from the re- 
quired text. The mass communication theory items focused on the rolft and 
~the effects cf modern communication media and historical development of 
contemporary media. The writing component also included a short essay 
question requiring students to differentiate the differences in writing w« 
style between oral and written media. 
, RESULTS 
Test Reliability 

Reliability estimates demonstrate that the ass-isment is a moderately 
reliable measure of test performance. Table 1 reports correlations among 
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the four part test with KR-20 reliability estimates on the diagonal. De- 
scriptive data revealed a mean score of 83.68 (S.D. - 8.57, Median - 85.50, 
Range - 96 - 62, S.E.M. - 1.62) or nearly a 70 percent average with the 
high score of 96 representing 80 percent. Similar statistics were observed 
with each test component; each yielded a mean score between 20. A and 21.9 
with scoring ranges from 10 through 27. The generally low scores may, in 
part, reflect the moderate reliability of the test. VJhen the thirteen least 
reliable items were deleted, the reliability increased to .80 with the mean 
score increasing to 79.79 or 75 percent and the median also increasing to 
75 percent. Furthermore, the public speaking component increased in relia- 
bility to .82 when the least reliable items were deleted. The remaining 
components also resulted in higher reliabilities, although none were as 
high as the public speaking component (i.e., .66 to .7A). 
Mediators of Test Performance 

Several variables were tested as potential mediators of performance. 
The majority of these did not affect scores (i.e., respondent sex, num- 
ber of transfer hours, number of major hours completed above the 36 minimum, 
academic minor, and academic scholarships awarded to students). 

One mediator in particular, grade point average (GPA), did influence 
performance on all four parts of the assessment. When overall grade point 
average was dichotomized above 3.12 (n - 19) and below 2.88 (n - 17), the 
consistent finding was that students with higher CPA's scored higher on all 
four test components and thus scored higher on the test composite (Low GPA - 
78.87, High GPA - 85.83, SS - 268.16, MS - 268.16, F - A. 10, p - .05). GPA 
was next treated as an interval-based variable by returning the dichotomized 
scores to their original raw scores. This procedure was selected so that 



8 



Assessment • 
8 

the strength of relationship between GPA and the assessment test could be 
determined. GPA correlated between .35 and .48 with all components except 
media writing (r - .21, p - ns). More specifically, as regression demon- 
strates, for every increase in GPA of one point, assessm«nt scores increas- 
ed by six points (MR - .44, R-squared - .194, B - .06, F - 6.52, p - .007, 

3 

Constant ■ 71.51). 

Two additional variables mediated test scores but did so less consis- 
tently than GPA. First, students completing an internship scored higher 
on the speech communication theory (Internship - 20.53, No Internship - 
18.27, SS/MS - 33.13, F - 4.91, p - .03) and the mass communication theory 
component (Internship - 20.80, No Internship - 18.09, SS/MS - 46.57, F - 
5.81, p - .02). While we cannot directly claim that the internship is 
responsible for higher test scores, it should be noted that the intern- 
ship is designed to help students sharpen, their theoretical understanding 
of communication as well as provide them an applied context in which to 
experiment with communication concepts and principles. A second mediator 
to affect assessment scores was full versus part-time employment. That 
is, students working part-time scored higher on the media writing component 
' (Part-time - 22.61, Full-time - 19.27, SS/MS - 22.34, F - 3.24, p « .08). * 
A final analysis entered all mediators in a regression model as pre- 
dictors of the test components md test composite. While none of the models 
yielded significance due to the large number of predictors tested, multiple 
correlations with the four components were all above .60 (R-squared - .36), 
thus suggesting that the mediators, collectively, partially explain test 
scores. Individual mediators did predict assessment scores when stepwise 
regression was attempted. For instance, the speech communication theory 

Q 



Assessment 
9 

component was predicted by number of years needed to complete the degree 
program (F - 3. 45, df - 10.36, part r - -.29, p - .07). This finding demon- 
strates that the longer students take to complete the degree, the lower 
their scores. Also, number of years needed to complete the degree (i.e., 
4-5 years vs. 6-10 years) yielded similar results with the oral competence 
or public speaking component (F « 7.06, df « 2.34, part r - -.39, p « .01). 
DISCUSSION 

Findings in this study offer several important xmplications on uhe 
role of indirect assessments on student learning. At the level of depart- 
luental advising, for instance, students should be encouraged not to seek 
full-time employment when enrolled in the media writing course, or .if 
needing to work full-time, be encouraged to enroll in the course if they 
can reduce the number of other courses they enroll. Currently, many stu- 
dents work full-time and enroll in an average of nine hours each semester. 
However, the two instructors who teach this course concur that the weekly : 
writing assignments require a commitment of time and energy to complete 
successfully and master basic writing principles needed in later course 
assignments. Students working full-time, although passing the course, 
^may not master writing principles because- of an already overloaded work 
and school schedule. 

Ideally, students should be encouraged to graduate in four years. 
However, this may not be practical for students who must finance much or 
, all of their education. The next best advice would be to wait as long 
as possible to take the required core courses to help guard against infor- 
mation loss. The underlying issue, however, is not retention of infor- 
mation since decay may occur whether students enroll in these courses early 
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or late in their coursework (i.e.. most students now complete the core their 
freshman year which may become problematic if requiring more than four years 
to graduate). Instead, the issue is whether students are provided an oppor- 
tunity to test their ..ew-£ourided knowledge once they complete a course. At 
the level of program administration, departmental policy may require stu- 
dents to complete an internship or practicum in which the application of 
course content is stressed. Should a required internship be administratively 
unmanageable or not deemed relevant to a student's degree program, other 
alternatives are available. For instance, a 1-3 credit senior seminar for 
graduating seniors may be more realistic. A seminar may include writing 
assignments to enhance writing competence and out-of-class speaking assign- 
ments to enhance oral competence. Another alternative may include compre- 
hensive testing in all courses so that students are encouraged to retain 
the information they have learned. 

Confirmation of these recommendations may require testing with addi- 
tional mediators. Universities that admit a large number of transfer stu- 
dents from nearby two- and four-year institutions, for example, also may 
wish to assess whether completing the required core elsewhere affects test 
^performance. Transfer credit was tested in this study but yielded too small 
a sub-sample (n - 6) for analysis. Comparison of overall GPA with depart- 
mental GPA also could be examined since overall GPA only accounted for 19 
percent of the variance in test scores. And, university entrance and 
'graduation requirements (i.e., standardized testing with normative tests and 
the breadth of required university core) are other candidates for testing. 
Examination of these variables may better determine whether the mediators 
tested in this study operate in conjunction with other untested variables. 5 
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A final factor not consistently controlled in this study also may have 
influenced assessment scores. Nearly one-third of each test component was 
constructed from text' material. These items would not be generalizable to 
institutions or courses not using these texts. Several of the items also 
may suffer a jargon-bias or a content emphasis bias and thus not reflect 
a generalized testing context as is recommended with assessment tests 
(Backlund, Brown, Gurry, & Jandt, 1982). On the other hand, assessment 
tests using select texts may be defended as partial predictors of aca- 
demic success so long as they are consisfint with the educational ob- 
jectives of an institution (e.g., preparing students for non-academic 
careers in the media or as diagnostic measures for assessing students' 
competencies before they enroll in advanced coursework) and are used in 
conjunction with direct assessments of copsmunication ability. The one 
test that best meets the requirements of assessment testing is the oral 
competence component; it is the most jargon- and text-free of the four 
components. 

Assessment tests are only one means of determining students' compe- 
tencies. On the other hand, GPA alone may not adequately reflect students' 

"ability to apply what they have learned in situations beyond the classroom. 
Findings reported in this study are particularly important in light of the 
fact that GPA did not yield significant differences with other mediators 
found to affect test scores. These findings strongly suggest that GPA alone 

' does not offset the effects of information loss while enrollment in an intern- 
ship does. More importantly, these findings also suggest that GPA may not 
improve students' ability to apply course content they have learned. Thus, 
the relationship between the cognitive and the behavioral components of 
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learning nay prove a fruitful line of research^ particularly in identifying 
if certain cognitive components are more important than others in influen- 
cing behavioral performance (e.g., if organizational skills are more im- 
portant than delivery skills). 

It should be emphasized that findings in this study may be more typical 
of non-traditional universities than other institutions in which fewer stu- 
dents work full-time or require more than four years to graduate. The fac- 
tors found to mediate assessment performance in this Mtudy may therefore hold 
more direct value for structuring the learning experience at a non-tradition- 
al university. While it may appear premature to report findings, for an 
assessment ba;. \? on only 36 students and a single year of nesting, important 
factors mediatl-;£ the learning experience at a non-traditional university 
have been isolated in u.iis study. Should additional testing confirm the 
cindir^:s reported in this study, indirect tests may be administered with 
reasonable confidence as diagnostic measures for placing transfer students in 
appropriate courses and as a screening test for determining the amount of 
theoretical background information to include in advanced courses (for 
students who have completed the required core course as many as three 
years earlier). 

In summary, this assessment test has yielded several instructi^'e find- 
ings regarding certain factors which mediate learning. These factors may, 
indeed, be as crucial to learning as the actual content that is learned. 
That is, the learning process itself may be strongly mediated by the format 
(i.e., number of years needed to graduate, student employment) through which 
students acquire their education. One of our goals as educators, particular- 
ly at non-traditional universities, should be that of identifying those 
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factors through which the educational experience may be most effectively 
administered. While many of these factors lie beyond our immediate con- 
trol (e.g., student employment), the factors which impede learning should 
be identified and counteracted through appropriate course design (e.g., 
comprehensive testing, reviewing basic content learned in core courses) 
and departmental procedures (e.g., senior seminar). Thus, one of the 
primary values of assessment, in addition to evaluating instructional 
effectiveness, in discovering additional ways in which we may best 
structure how the educational experience is processed. 
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^ Footnotes 

^Pour studentvi completed the speech communication assessment— too small 
a number for comparison to the 36 who completed the mass communication assess- 
ment- However, analysis of overall scores revealed descriptive statistics 
similar to students taking the mass communication test. 

copy of the 20-page mass- and speech communication booklets are 
available from the author. 

^GPA was entered as a whole number without decimals in the model, 
thus explaining the B term in the regression model. 

Univariate analysis of variance was selected over the more stringent 
multivariate analysis of variance in order to identify trends within the 
data so that preliminary decisions may be made regarding mediators to in- 
clude in future assessments. However, multivariate analysis approached 
significance with GPA (p « .07). 

^Assessment scores could not be compared with the COHP because those 
results have not been made available, as yet, by the university. 
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Oral Comm. Theory 


Mass Theory 


Writing 


Composite 


Oral 


(.71) .47 


.31 


.51 


.74 


Comm. Theory 


(.61) 


.30 


.47 


.71 


Mass Theory 




(.66) 


.52 


.48 


Writing 






(.66) 


.74 


Composite 








(.75) 



*p » .01 level or higher 
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